Information systems may rely on data obtained from a variety of information sources. For example, an information system may store or index data obtained from various content providers, such as map data from a geographic information service provider, phone number and address data from a telecommunications service provider, news data from news service providers, or any other data from other types of content providers. An information system may also store or index data generated by actual users of the information system. For example, user-generated data may include photos, videos, product/service reviews, blog postings, or any other data generated by users.
The value of an information system is based at least in part on the accuracy, relevancy, and utility of the data that the information system stores and/or indexes and ultimately provides to a user. For example, if a user receives or finds inaccurate, irrelevant, or offensive data via a particular information system, the user may lose faith in that particular information system and may choose to obtain information from different information systems in the future. Conversely, if a user consistently finds or receives accurate and relevant data via a particular information system, the user is more likely to continue using that particular information system.
Therefore, one goal of an information system operator is to ensure that the data stored in or indexed by the information system is accurate, relevant, or otherwise useful and not offensive. One way to increase the accuracy and relevancy of data stored in or indexed by an information system is to review or moderate the data stored in or indexed by the information system. One way to moderate the data stored in or indexed by an information system is to have humans review the data to determine whether the data is accurate, relevant, or otherwise useful and not offensive. However, having humans review all the data stored in or indexed by a particular information system may be challenging if the information system stores or indexes large volumes of data because of the large number of human reviewers that would be required to review all the data.